Dynamic Causal Monitoring for Distributed Systems

Monitoring and troubleshooting distributed systems is notoriously di›cult; potential problems are complex, varied, and unpredictable. e de-factomonitoring and diagnosis tools at our disposal today – logs, counters, and metrics – have two important limitations: what gets recorded is dened a priori, and the information is recorded in a componentor machine-centric way, making it extremely hard to correlate events that cross these boundaries. is report is an extended version of our full Pivot Tracing paper [68]. Pivot Tracing is a monitoring framework for distributed systems that addresses both limitations by combining dynamic instrumentation with a novel relational operator – the happenedbefore join. Pivot Tracing gives users, at runtime, the ability to dene arbitrary metrics at one point of the system, while being able to select, lter, and group by events meaningful at other parts of the system, even when crossing component or machine boundaries. We have implemented a prototype of Pivot Tracing for Java-based systems and evaluate it on a heterogeneous Hadoop cluster comprising HDFS, HBase, MapReduce, and YARN. We show that Pivot Tracing can ešectively identify a diverse range of root causes such as soŸware bugs, misconguration, and limping hardware. We show that Pivot Tracing is dynamic, extensible, and enables cross-tier analysis between any inter-operating applications, with low execution overhead. is report extends the original paper’s discussion [68] of Pivot Tracing’s implementation and provides further details on instrumenting and operating Pivot Tracing within distributed systems.